The Impact of Including 1 Running Head: The Impact of Including The Impact of Including Predictors and Using Various Hierarchical Linear Models on Evaluating School Effectiveness in Mathematics

2-level, 2-level gain score, and 3-level hierarchical linear models were fit to data containing 2 years of standardized test results from a large Midwestern urban school district. First the models were fit including only initial score as a student level predictor of final score. Then, each model was fit adding race, socioeconomic status, and race and SES as student level predictors. Third, each model was fit adding race, SES, and race and SES as student and school level predictors. School effectiveness rankings varied by the type of model but not the predictors that were included. The 2-level gain and 3-level models were most valid; however, because of its power and flexibility, the 3-level model is best for examining school effectiveness. The Impact of Including 3 The Impact of Including Predictors and Using Various Hierarchical Linear Models on Evaluating School Effectiveness in Mathematics A fair and accurate way to assess the effectiveness of individual schools is to determine how much value a particular school adds to student learning by examining student progress in their individual achievement levels over time (Andrejko, 2004; Crane, 2002; Kupermintz, 2003; Olson, 1998; Popham, 2005; Tekwe, et al., 2004). Value added accountability systems can be used to more accurately identify which schools are failing or succeeding, with success or failure being determined by student growth over time, as opposed to the change in proficiency classification rates for different cohorts of students over time (Crane, 2002). These value added accountability systems permit schools to be evaluated based on how much progress their students have made, regardless of their achievement levels when they entered the school (Ballou, et al., 1999; Raudenbush, 2004a). They provide a way of recognizing that schools serve students who start at different places and progress at different rates (Heck, 2006). Hierarchical Linear Models One particular way to implement a value added accountability system is to make use of hierarchical linear modeling. By its very nature, research pertaining to the effectiveness of a school requires exploration of hierarchical data because students are nested within classrooms and schools (Raudenbush & Bryk, 1986). Hierarchical models simultaneously model student level relationships and take into account the way students are grouped into individual schools and/or classrooms (Goldstein, 1997). By modeling both the student level and the school level simultaneously, the annual academic growth of students is separated into that which can be attributed to the student and that which can be attributed to the school (Hershberg, 2005; Meyer, 2003; Raudenbush & Bryk, 2002; Yeagley, 2007). Therefore, hierarchical linear modeling is a The Impact of Including 4 way to isolate the impact of instruction on student learning. One advantage of using hierarchical linear modeling is that these models allow for improved estimation of effects within individual units, such as schools, which is accomplished through Empirical Bayes estimation. Empirical Bayes estimates are also called shrinkage estimates because in these estimates the group mean is pulled toward the grand mean proportionately to the group sample size and the reliability of the group mean (Raudenbush & Bryk, 2002). Several different hierarchical linear models can be used for these purposes and there have been no empirical studies examining which model is best. Thus, the current study compared and contrasted the results obtained from 2-level, 2-level gain score, and 3-level individual growth hierarchical linear models to determine which model is best when trying to ascertain the effectiveness of a school. According to Meyer (1995), achievement growth from one grade to the next can be adequately characterized by a 2-level hierarchical linear model, with student characteristics represented by level-1 of the model and school characteristics represented by level-2 of the model. In a model such as this, a student’s final achievement test score is entered as the dependent variable of the level-1 model and the initial score on the test is entered as one of the independent variables in the level-1 model. Then, each of the regression coefficients in the level1 model are conceived as outcome variables of the level-2 model that are hypothesized to depend on specific organizational characteristics. For example, if the initial achievement test score is included as a predictor in level-1 it becomes an outcome variable in one of the level-2 equations. Level-2 (school level) predictors such as race and SES could then be included as predictors of initial score. A unique set of predictors can be specified for each regression coefficient. These level-2 predictors show the net effects of the predictor after controlling for all of the level-1 predictors (Raudenbush & Bryk, 2002). The Impact of Including 5 In general, the equations for the 2-level model are as follows: